The phrase "pumpkin haters" refers to people who strongly dislike or have a negative attitude towards pumpkins.